Repository navigation
fix: lone high surrogate decoded as U+0000 instead of an error - #482
Conversation
|
At In if !LOW_SURROGATES.contains(&code_point_2) {
return Err(ErrorType::InvalidUnicodeCodepoint);
}
|
get_unicode_codepoint returned Ok((0, _)) for a high surrogate that is not followed by a `\u` escape, and for a surrogate pair with invalid hex digits. The 0 is a leftover sentinel: when this code returned the number of bytes written, 0 meant failure and callers checked `o == 0`. Since 29d9998 it returns the code point instead, so the 0 became U+0000 and "\ud800" parsed as "\0" without an error. Return InvalidUnicodeCodepoint in both places, and add a regression test covering the lone-surrogate forms (fails before this change). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
0e253fb to
6121944
Compare
|
Thanks @symbiogenesis for the report — confirmed: af7f553 checks the second escape against |
get_unicode_codepoint checked the second escape of a pair only against the lower bound (checked_sub(0xdc00)), so "\ud800\ue000" decoded as U+10400 where serde_json rejects it. Check the second escape against LOW_SURROGATES before combining; invalid hex digits set the high bits and fail the same check. The test now covers both bounds of the low-surrogate range, the serde path, and the first and last valid pairs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7348eb5 to
af7f553
Compare
|
thanks @ddxd |
Fixes #481.
get_unicode_codepointreturnedOk((0, src_offset))for a high surrogate not followed by a\uescape, and for a surrogate pair with invalid hex digits in either half. The0was the old "bytes written = 0 means failure" sentinel fromhandle_unicode_codepoint. Since 29d9998 the function returns a code point, so it now means U+0000 and"\ud800"parsed as"\0".This change returns
Err(ErrorType::InvalidUnicodeCodepoint)in both places. Everyparse_strimplementation already maps an error from here toInvalidUnicodeCodepointat the escape's position, so no caller changes are needed.Test:
lone_high_surrogate_is_an_errorcovers"\ud800","\ud800x",["\udbff"], a surrogate inside a string,"\ud800\n","\ud800\uzzzz","\ud800A"and"\udc00"through bothto_tapeandto_borrowed_value, and checks that a valid pair still decodes. It fails without the fix (accepted: "\ud800") and passes with it.cargo testpasses on aarch64 (macOS, NEON) and x86_64 (Linux, AVX2), andcargo fmt --checkis clean.Side note, not changed here: the
if o == 0checks afterhandle_unicode_codepointin the avx2/sse42/neon/simd128/portableparse_strcan no longer trigger, becausecodepoint_to_utf8writes at least one byte. I left them in to keep this diff minimal. Happy to remove them in a follow-up if you prefer.🤖 Generated with Claude Code